NeRFs For Capturing Art
by Tobin Cavanaugh
Neural Radiance Fields (NeRFs) are some of the coolest up and coming tech in the world of rendering and spatial capture. Photogrammetry is decent, but NeRFs fill a new niche, that being having detailed 3D captures that include material data and light simulation.
The solution to preserving art digitally in the modern day are NeRFs. I recently went to the Prada Museum in Madrid, Spain. It was a fantastic experience, and after two and a half hours my head and feet hurt from seeing so much art. I didn't get to see everything I wanted to see there, and when I went afterwards to look up photos of certain art pieces, I couldn't find good quality ones. Photos aren't allowed in the Prada Museum (though I took a few sly ones) so my memory of the art is the only way I can experience a lot of that art again.
The solution is NeRFs, by scanning these art pieces with high resolution cameras and outputting a high resolution NeRF, we could open up some of the greatest works of art ever to people who aren't able to go to the actual museum. I would love to go back to the Prada Museum, but I just don't have the money or time, and I think most people are in the same boat. Opening up this art to the rest of humanity would only have good effects, and I think it would make the world a better place.
Why NeRFs in particular? Besides the fact that the tech is convenient, NeRFs maintain things that would normally be lost by regular photogrammetry or regular photography. An obvious one is the brush strokes and how they reflect the light. Most paintings are not purely 2D, most have brush strokes and paint that lift off of the canvas, leading to a deeper view of the painting. When a 2D photo or even a photogrammetry scan is taken, this is lost. The frames are also something that is lost by 2D photos. Many paintings I saw had unique framing schemes, particularly Bartolomé Bermejo's Saint Dominic of Silos enthroned as a Bishop. Along with good results, NeRFs are also somewhat technologically agnostic, in the sense that the raw data is just photos, there aren't any special cameras or formats used. This means that the old raw photos could be preserved, and should NeRFs be superseded, the data could easily be retrained and users could be served a new, better, view.
These are from Wikipedia and the Prado Museum Website, respectively.
As you can see the same painting with the frame included and without it included are very different experiences. Both of these, however, are still 2D photos, and don't at all capture the true scale and 3D of the ornate frame. The best solution to expressing the entirety of the painting is via a NeRF. The frame, in all its intricacy, would be represented fully and faithfully.
As I mentioned earlier, the loss of a dimension is more significant than one might immediately assume. Beyond just the frame and whatnot, we also lose all of the experience of the light. An important one to me is the Vista de Bermeo by Paret y Alcazar Luis, which you can see in my photo below. A similar image can also be found at the Museo Del Prado Website.
 |
|---|
This is not a painting, but rather a type of mosaic named Commesso or Florentine mosaic [1]. What's lost in this image is the way the light reflects the stone. At a distance like the photo, the piece almost looks like a painting, but at a glancing view, you see the sheen of the individual stones. Each type of stone, while smooth, has a distinct look at this angle.
This is entirely lost in the photo, in fact it would require numerous different photos at different angles to express. The solution, of course, is NeRFs.
A great demo of what this tech can achieve in the lighting space is on Matthew Tancik's blog. This can be seen in the following video, where the TV screen reflects the light accurately. The right side demonstrates that the light can be viewed independently of the camera angle.
